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PREFACE 



This Report contains the text of an Evaluation Manual prepared 
in response to the need of the Deputy Assistant Secretary for Evalua- 
tion and Monitoring of the Department of Health, Education and Welfare 
for such a manual to aid in building an effective evaluation structure 
throughout the agencies of HEW. 

The HEW request was for a guide not to the techniques of evaluation, 
but to the purposes of evaluation and how evaluation programs could 
meet these purposes. For this reason, the manual tries insofar as is 
possible to stay away from questions of methodology, concentrating 
instead on the decisionmaking context for evaluation. 

Nor does the manual attempt to lay out rules for evaluators which, 
if followed, would get them to do evaluations "right. Conceptually , 
it might have been possible to provide decision rules for likely con- 
tingencies at a level of detail such that evaluators at all levels of 
HEW would have carried out relevant and competent evaluations simply 
by following those rules. In reality, however, this is quite impossible 
perhaps it never will be possible. Rather than setting forth such rules 
then, the text is written so that if an evaluator follows it he will 
provide explicit answers — for the record — to the questions that should 
be asked by those who plan and carry out evaluations. Such answers 

not in themselves produce good evaluations, although they should 
help by directing the evaluator to Important considerations he might 
otherwise neglect. Even more important, hov;ever, such on-the-record 
answers to the relevant questions will enable administrators and over- 
seers of evaluation to determine whether the kind of evaluation they 
want is being done, and to institute the necessary corrections if they 
feel that it is not. A manual that provides this sort of basis for 
further action is less ambitious than a manual that in Itself directs 
what should be done; the latter, however, would be neither usable nor 

used. 

The manual has been prepared under a subcontract with Carnegie- 



Mellon University. 
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I. INTRODUCTION 



Evaluation, in its most general meaning, is a process of measure- 
ment designed to estimate worth. The definition does not imply any 
answer as to worth is being estimated. As viewed from a government 
evaluation office, however, the "why" question is central, and a narrower 
meaning must be given to the word "evaluation. Evaluation of govern- 
ment programs or projects is a process of assessment designed to pro- 
vide information about past and present operations and effectiveness , 
in order to assist in malting decisions about the future . 

Within the Federal government, evaluation offices often have addi- 
tional responsibility for activities that, while related to evaluation, 
are distinct from it. Within the Department of Health, Education and 
Welfare, evaluation offices include among their activities the develop- 
ment of information or data systems and some monitoring or compliance 
control activities. Since these activities often compete with evalua- 
tion for the same funds, they will be considered explicitly in this 

Guide. 

Because the importance of evaluation to the government is in the 
assistance it gives to decisionmaking, it must be planned and designed 
with the decision function in mind. In addition to being technically 
competent , government program evaluation must thus be relevant to 
decisionmaking and decisionmakers, and it must be credible. Credibi- 
lity is based in part on technical competence and in part on the 
evaluator's reputation for independent appraisal. 

This is not a Guide on how to evaluate. It is, rather, a Guide to 
assist evaluators in the development of a strategy: in determining 

when to evaluate, who should evaluate and supervise the evaluation, 
how to allocate efforts among different types of evaluation; and in 
assessing evaluation. The Guide has four objectives: 

1. To aid in the planning of evaluation and related activi- 
ties so that the limited resources available to evaluation 
offices are put where they are most needed. 
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2. To aid in carrying out evaluations in ways that are most 
likely to help in the making of decisions. 

3. To aid in assessing and improving the total evaluation 
effort. 

4. To aid in using evaluation as a part of the policymaking 
process. 

Section II below presents a typology of evaluations a mode of 
classification that can be used as an aid to the evaluation planning 
and execution processes with which the Guide Is primarily concerned. 
Section III covers the planning of evaluation; Section IV, the prepara- 
tion for specific evaluations; and Section V, the preparation of Cri- 
tical Program Summaries, which brings together the evaluator's views 
on the programs he is responsible for. An appendix summarizes the 
documentation called for by this Guide. 
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II. TYPOLOGY OF EVALUATION AND RELATED ACTIVITIES 

Evaluations can be classified in a number of ways; by what Is 
being evaluated, by who conducts the evaluation, by the decision that 
is supposed to be affected by the evaluation, by the methodology used. 
The appropriate classification depends, of course, on the purposes for 
which it is to be used. The classification here provides a beginning; 
undoubtedly its use will indicate needed changes. 

This typology is designed to be used In planning an evaluation 
program, to provide summary descriptions for planning the allocation 
of available evaluation funds and efforts. It should also be used In 
the preparation of a specific evaluation, as a means of shorthand 
notation on the functions of the project. In each case, it is neces- 
sary to indicate the purpose of the evaluation, what Is being evaluated, 
and for what decisionmaking official the evaluation Is being made. 

These three dimensions of the typology are discussed below. 

A. FOR WHAT PURPOSE? 

Evaluation and monitoring funds within the Department of Health, 
Education and Welfare are expended in the service of three broad ob- 
jectives: evaluation proper. Intended to aid in making decisions about 

a program or a treatment within a program; monitoring, more accurately 
described as compliance control, and building of capabilities for 
further evaluation. The distinctions among these categories, and among 
their subcategories — particularly the two major subcategories of eva- 
luation proper — depend in large measure on wh^ the evaluation project 
is being planned or set up. The key is the decision it is hoped the 
project will affect: if the project is designed to have impact on a 

decision concerning program size or program strategy, it belongs in 
one of the subcategories of evaluation proper; if it is to check ad- 
herence to regulations, it is compliance control; if its aim is not to 
contribute directly to program functioning, but rather to improve 
future evaluation efforts, it should be classed as capability building. 
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1. Evaluation 

a. Evaluation to Affect Resource Allocation . This Is the classi- 
cal kind of evaluation designed to assess the worth or effectiveness 
of an ongoing program or project with a view to helping determine the 
quantity of funds or other resources It should receive. In the limit, 
the choice may be between no resources and some — a "go/no-go” decision 
to continue or cut off a project or program. More ordinarily, however, 
the decision based on such an evaluation Is one of resource allocation 
at the margin — putting more Into the "best" programs, holding back or 
cutting back on the less worthy. 

Evaluations In this category vary with respect to the basis of 
comparison used In assessing relative worth: the Impact or effective- 

ness of a program may be compared with unmet needs, with specified per- 
formance objectives, with past program performance, with the perform- 
ance of equivalent programs, and/or with Its own costs. Comparison 
with equivalent programs on the basis of program costs and of results 
measurable In the same units (e.g., percent of students In an In-school 
program whose reading; scores have been raised by one grade level as 
compared with the same percent for program carried on outside the 
regular school system) Is sometimes termed cost-effectiveness evaluation. 
The Implicit decision to which such an evaluation Is addressed Is that 
of allocation among the programs whose costs and effectiveness are 
being compared. If program results can be expressed In dollar terms 
comparable not only to other program results, but also to program costs 
(e.g., the dollar gains 1:*. Increased earnings attributable to a voca- 
tional rehabilitation program, as compared to the dollar costs of the 

program), benefit-cost analysis resulting In a difference between bene- 
* 

fits and costs (expressible In a single number) Is possible. Then 
the Implicit allocation decision Is between the program being evaluated 
and all other uses of federal dollars (Including lower taxes) . 



More traditional practice has been to calculate the ratio of 
benefits to costs, but recent thinking suggests that the difference 
provides more Information concerning alternative uses for the program 
funds In question. 
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In fact, meaningful benefit-cost computations are seldom feasible, 
but allocation to the program being examined as against other uses of 
the federal dollar is the Implicit basis for many evaluations. The 
effort is to examine what charge — if any — against the agency budget, 
the Department budget, or the Federal budget is warranted. (See C, 

below "For Whom" is the evaluation being made?). Evaluations whose 

primary purpose is to justify a program to higher authorities (if the 
evaluations come out right) are of this type, since the decision to be 
made--by the Office of Management and Budget or a Congressional com- 
mittee, or whomever the program is being justified to—is a decision 
about allocation to the program as against a range of other programs. 

b. Evaluation to Improve Strategies . This is the kind of evalua- 
tion designed not to see how well a program is doing as a whole, but 
to determine changes (alternative strategies) that will make it perform 
better. Such evaluations provide data for making decisions about pro- 
gram development and program revision. They may be designed to provide 
a basis for choice among techniques to be used within a project or 
program; or they may be designed to support technique refinement in an 
iterative process of evaluation and marginal change. Technique choice 
evaluations make a summary judgment about the relative effectiveness 
of new or existing alternatives, in a way similar to the allocation 
evalviatlons of whole programs discussed in (a) . Technique refinement 
evaluations give the decisionmaker more or less continual feedback on 
the effectiveness of particular aspects of a project or program when 
it is in the formative stage. 

Categories (a) and (b) are not mutually exclusive; a particular 
evaluation project may provide information of both types. Measurement 
of the effectiveness of Head Start in improving the learning capabili- 
ties of pre-school children, for example, may be designed to affect the 
allocation of anti-poverty funds to Head Start. However, if it compares 
Head Start effects on pre-schoolers of different ages in order to esti- 
mate the best age for initial intervention, it is also a strategic 
Improvement evaluation dealing with technique choice. 
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In deciding whether an evaluation project or part of an evaluation 
project should be classified as an "allocation" or an "Improvement" 
evaluation, the decision level for which the Information Is Intended 
must be taken Into account. For Instance, an evaluation project might 
be designed to help project managers determine what makes the best 
projects so good; If so. It would be an Improvement evaluation. Or It 
might be designed to help program managers determine which projects 
should be cancelled; if so. It would be an allocation evaluation. 

While In a brief description the two evaluations might sound very 
similar, they would In fact be quite different If appropriately de- 
signed to meet the differing decision needs. The one designed for 
Improvement would concentrate on the mix of techniques within the 
projects and on environmental conditions confronting good and bad 
projects. The object would be to find out what techniques work best 
under what conditions, with a view of spreading the use of the best 
techniques under the appropriate conditions. The one designed to 
determine relative worth would concentrate primarily on the worst 
projects, and would attempt to compare the costs of cancellation with 
the costs of continued operation. It will frequently be economical, 
however, to look carefully at both sets of factors In order to provide 
Inputs for both types of decisions. 

2. Compliance Control 

These activities, often called monitoring, are distinguished from 
evaluation proper by their attention to Input measures (e.g., adherence 
to guidelines, workloads, administrative practices.) Such measures are 
different from the estimates of worth which are central to any defini- 
tion of evaluation. The term "compliance control" Is more descriptive 
of this category than "monitoring"; It Is possible to monitor perform- 
ance, but the activities normally referred to here monitor compliance 
— compliance with legislative Intent and administrative regulations 
— not performance. 

Under compliance control, two kinds of activities deserve separate 
mention: 
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a. Site visits by inspectors. (Not all site visits are 
for compliance control, however. Many evaluations, 
fitting within the strictest definition of the term, 
will include subjective components — estimates based 
on the close observations of trained evaluators.) 

b. Routine reporting of input and workload data. 
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3. Capability Building 

An evaluation office may engage in a number of activities designed 
to facilitate or improve the conduct of evaluation studies in the future. 

refer to this sort of effort as "capability building." Three sub- 
categories can be listed: 

a. Developing a Data Base . Some data are useful for future 
evaluation activities, other data are not. While there will always be 
questions as to whether an investment in a data system should be listed 
under (2b) or (3a), it is important that relevant issues about such an 
Investment be raised. One such issue is the distinction between collect- 
ing data for administrative use and collecting data for use in future 
evaluations. Another major issue is whether data systems should be set 
up now for evaluations to be carried out in the future, or whether ad 
hoc data collection efforts within future evaluations would be more 
efficient and economical. 

b. Improving Evaluation Strategies and Methodologies. This cate- 
gory includes all activities directed at improving the state of the 

art. 

C. ImoroYlia lton-f.J«ral Cpabllltles . This category Includes 
all activltias directed at dissanlnatlng the state of the art of other 
level, of government In order to help state end local decisionmakers 
—and private groups as well— to Increase their abilities to carry out 
their own evaluations. One of the most Important roles of HEW Is as 
a service org«Hretlon to public and private entitles outside of the 
federal government. These groups have a major role In evaluating 
federal progr<ms as they apply locaUy. and a major need to evaluate 



15 






- 8 - 







i 



o 

ERIC 



their own programs, and HEW assistance can aid them in carrying out 
effective evaluation. 



B. WHAT IS BEING EVALUATED? 

The types of activities that might be evaluated can be classified 
as follows: 

1. Major separable program (s) ; e.g.. Title I of the Elementary 
and Secondary Education Act, Maternal and Child Health Care, 
Public Assistance. 

2. Major component(s) of such a program; e.g., urban school 
assistance, prenatal health care. Aid to Families with 
Dependent Children (Unemployed Parents) . 

3. Operating project(s) within one of these programs; e.g.. 

Title I assistance in Chicago. 

4. Pilot or demonstration project(s); e.g.. Demonstration 

Neighborhood Health Centers. (The evaluation of pilot or 
demonstration projects which is frequently within the 
mandate of an evaluation office, is Included here: the 

setting up of such projects is not Included within this 
listing. Evaluation of a pilot or demonstration may be 
used for purposes of allocation — should the program being 
tried on a pilot basis be given the resources to become 

a large-scale effort? — or for improvement — should the 
technique being tried be replicated throughout an 
existing program?) 



C. FOR WHOM? 



Different kinds of evaluations are useful at different levels In 
the decision structure, as the example on the relative effectiveness 
of projects Illustrates. 



For most purposes, the relevant decision levels are, in descending 
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The Secretary of Health » Education and Welfare 

2. The Surgeon-General, the Commissioner of Education, the 
Administrator of the Social and Rehabilitation Service, 
the Commissioner of Social Security 

3. Bureau Chiefs 

4 . Program Managers 

5. Project Managers 

6. Non-federal Decisionmakers. As noted above, public and 
private groups outside of the federal government have a 
need for evaluations of their own programs. In many 
cases, the most efficient way to carry out such evalua- 
tions is for federal evaluators to execute them. Par.uic- 
ularly in the case of improvement evaluations, HEW may 
want to evaluate a new technique, which, if the evaluation 
is positive, will be useful across a broad range of 
programs and localities. 
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III. ANNUAL EVALUATION PLANNING 

An evaluation plan lays out in advance the activities that an 
office responsible for evaluations expects to carry out during the 
period for which the planning is being done. In preparing such a plan, 
the responsible office can allocate the resources available for evalua- 
tion““both dollars and skilled manpower— In a way that puts first things 
first and that does not inake commitments to relatively low priority 
evaluation efforts before funds are set aside for the most important 
ones. In addition, the evaluation plan provides those to whom the 
evaluation office is responsible with an opportunity to review projected 
evaluation efforts in the light of broader sets of needs. An evaluation 
plan Is not a straitjacket, it is a guide. It should be considered 
changeable— because priorities have changed, because possibilities have 
changed, because new evaluation needs have been identified, ^ut the 
plan Is a discipline, and an aid to self-discipline. When new needs 
do arise, consideration will have to be given to whether they^ can be 
filled by obtaining additional resources for evaluation, or whether 
previously planned lower priority items will have to be dropped or 
postponed . 

The planning formats and tabular summaries laid out here are de- 
signed to be useful to decisionmakers and to evaluators, to be simple 
and clear, and to minimize the duplication of planning efforts. They 
minimize the back and forth flow of paper by confining routine flows 
to summaries and brief abstracts. These short documents can then be 
used to flag the availability of more comprehensive statements retained 
in the files of the preparing offices and available upon request to 
responsible administrators and higher evaluation echelons. 

To these ends, what Is suggested is an annual evaluation plan, 
timed so that the preparation and review of the plan will be completed 
before it is necessary to make commitments to the planned projects. 
Although evaluation and evaluation planning must be matched to other 
time-cycles as well — the budgeting cycles of the programs being eval- 
uated, for example — this can be accomplished by preparing different 




18 



- 11 - 



slmple summaries of the single plan, rather than additional plans. 

The summaries will also facilitate after-the-event review of the eval 
uation plan. At the end of each year, the summaries of the evaluations 
planned can be compared with summaries of evaluations actually carried 
out, to discover how effective evaluation planning actually was. 

To achieve this, then, each office responsible for evaluation 
should prepare an annual plan consisting of three par_t .^; 

0 A statement of the strategic objectives of the evalua- 
tion plan. 

0 A brief on each planned evaluation, to be prepared otily 
for those evaluations for which the office has primary 
responsibility . 

o A number of aggregated tabular summaries, covering both 
the evaluations for which the office has primary respon- 
sibility, and those for which it has overall review 
responsibility . 

A. STATEMENT OF STRATEGIC OBJECTIVES 

The statement of strategic objectives should specify the programs 
and program areas for which the plan-preparing office has the primary 
responsibility for initiating, contracting, and supervising evaluations; 
and those for which it has the review responsibility for overseeing 
evaluations developed by offices in subordinate agencies. 

For example, the Assistant Secretary for Planning and Evaluation 
has the primary responsibility for developing evaluations cutting 
across more than one HEW area (e.g,, education and vocational rehabili- 
tation), and also for carrying out specified evaluations for the Secre- 
tary. In addition, he has review responsibility for evaluations de- 
veloped by the evaluation offices of the Office of Education, the 
Health Services and Mental Health Administration, and so on. 

The statement should then lay out in the light of these respon- 
sibilities, the evaluation strategy for the year being planned. It 
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should list by rough priority ranking the programs and program areas 
to be stressed in the year's evaluation; it should give the reasons 
for the ranking in the light of the program decisions that must or 
should be made; and it should explain how the decisions may be affected 
by the evaluations. In addition to specific program evaluations, the 
statement should discuss and put into the ranking any compliance-control 
projects (A2 in the typology) for which the office is responsible, and 
any capability-building efforts (A3) — important for long-run program 
and evaluation purposes — that should be initiated during the planning 
year. The statement should discuss in general terms the kinds of 
evaluations that will be carried out to meet the objectives, their 
scope (e.g., single program or cutting across more than one program), 
their method, and their timing, and should make clear for which ones 
the office preparing the plan has primary or review responsibility. 

B. EVALUATION BRIEFS 

The plan should Include a separate brief on each evaliiatlon for 
which the office preparing the plan expects to take primary responsi- 
bility. A brief prepared as part of the plan should ordinarily list: 

1. The program or program area being evaluated. 

2. The evaluation category in terms of the typology of 
Section II, above. 

3. A short description of relevant policy decisions and 

of the ways in which different possible outcomes of the 
evaluation may affect the decisions. 

4. The timing of the evaluation: when it is expected to be 

initiated and on what schedule the information it is de- 
signed to produce is likely to become available. 

5. An estimate of the nximber of dollars expected to be 
obligated in the Initial year and in subsequent years. 

6. An estimate of the number of man-months of evaluation 
office supervision required in the initial and in 
subsequent years; and. If possible, the name of the 
initial supervisor. 
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In addition to these briefs — which should In fact be brief — the 
specific evaluation rationale suggested In Section IV of this Guide 
for use In preparing any single evaluation, and the after-evaluation 
analysis also suggested there should be filed In the preparing office 
with the relevant brief, as the rationale and analysis become available. 
These rationales and analyses should, of course, be available to admin- 
istrators and other evaluation offices In the supervisory chain; but 
to cut down on the flow of unnecessary paper, they should not be for- 
warded routinely^ Any arrangements made on review of amendments to 
the evaluation plan should be decided by the relevant offices. 

C. TABULAR SUMMARIES 

As noted, the Individual evaluation briefs should be Included In 
the plans only of the offices with primary responsibility for these 
evaluations. Frequently, however, an evaluation office not only has 
direct responsibilities, but also review responsibilities over the 
evaluations carried out by a number of other offices. The simplest 
way to fulfill the planning requirements Inherent in such overall res- 
ponsibilities Is to tabulate all the evaluations for which the office 
putttlng out the plan has primary or review responsibility. 

1. Basic Format 

Sample Table 1 provides a basic format for such tabulations. By 
listing a year's evaluations vertically by program or component being 
evaluated, and horizontally by purpose of evaluation, and summing both 
ways, it provides an easy but detailed picture of allocation of effort 
oy major categories. The completion date and the decision level for 
which the evaluation Is being conducted are also Indicated In this 
table. 

An Individual evaluation project may fit Into more than one cate- 
gory; e.g., an evaluation may (and frequently. should) be designed both 
to affect resource allocation, and to improve program strategies. In 
these cases, the dollars allocated to a particular project should be 
apportioned across two (or more) relevant categories. Hypothetical 
project 867 provides an example In Table 1. 





Sample Table 1 

(EVALUATION LISTING) 
Planning Year (e.g. 1972) 
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Each annual evaluation plan should Include separate tables of 
this type on: 

a. Planned activity in the year for which the plan has 
been prepared (e.g., 1972, in Sample Table 1). 

b. Best estimate of actual activity in the current year 
(during which the plan has been prepared) (e»g>, 1971). 

c. Actual activity in the last complete year (e.g., 1970). 

d. The plan that had been laid out for the last complete 
year when it was the planning year (e.g., the plan 
prepared in 1969 for 1970) . (This table of course 
cannot be Included until two years after the system 
suggested here has been adopted. Its purpose Is for 
comparison to the table suggested In (c) , to examine 
how closely the evaluation plans are actually being 
carried out.) 

2^j Summary Tables 

Sample Table 2 provides a format for a series of planning summaries 
of use both to the evaluation office and to various review levels. 

Without listing each separate project as was done In Table 1, Table 2 
tabulates vertically the total evaluation effort by major program and 
by type of evaluation (or compliance control or capability building) 
(Breakdown A In the typology of Section II.) This can be done on a 
single sheet for a number of years, and can Include both dollar and 
manpower allocations, useful for different sorts of allocation. 

Tabulations In the Table 2 format can be adapted and used flexibly , 
summarizing In different ways for different purposes. Initially, the 
. following three versions of Table 2 are suggested for Inclusion In the 

plan: 

a . A summary of the evaluations to be Initiated during the plan- 
ning year (and other years). For the planning year, the total of 
dollars obligated should equal the total available for obligation. 

(It Is perfectly legitimate, of course, to Include an unallocated con- 
tingency fund.) 
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b. A summary of the evaluations that V7ill be on-golng during the 

planning year and subsequent years. This Is a cumulated level-of-activ- 
Ity table: for the planning year It should include both evaluations 

begun in previous years that will still be on-golng, and evaluations 
expected to begin in the planning year. For subsequent years, it should 
carry the implications of current and past plans for evaluations to be 
on-golng then. For the planning year, the total of man-months of super- 
vision to be used should be no greater than the total staff-time avail- 
able for this purpose, allowing again for contingencies. 

c. A summary of the Information that will becom e available during 
the planning year and subsequent years. In this summary, evaluations 
should be tabulated by the year in which the final report is expected 
to be available; if interim reports providing enough information to 
affect decisions are also expected, they should be tabulated also. 

Since reports may be listed more than once, column totals in this 
context are not likely to be very meaningful. 

For each of these tables, the first year listed should be the 
year completed before the date of the plan; actual experience should 
be tabulated in that column. For the year during which the plan is 
being prepared, the Information should be based on best estimates. 

How far beyond the planning year a table should be carried depends on 
the purpose of the table in question. A summary of evaluations to be 
initiated may go no further than the planning year, although if con- 
sideration has been given to subsequent years, additional columns may 
be useful. A summary of evaluations on-golng should be carried out 
for as many years as any current evaluations are expected to continue; 
similarly with a summary of information to become available. 

The purpose of tables like these, then, is twofold; 

o To facilitate easy reference to Important data about the 
course of evaluation and its expected effect on decisions. 

o To allow evaluation offices with review responsibilities 
to oversee these responsibilities simply by constructing 
broad aggregate summaries from the narrower summaries 
provided them by thfe offices being reviewed, rather than 
by duplicating the entire planning process at every level. 

; 25 
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IV. THE RATIONALE FOR A SPECIFIC EVALUATION 



This section Is aimed at assisting evaluation offices, first, to 
lay out In advance the objectives of a proposed evaluation, and, 

second, to assess in retrospect how well the evaluation met these ob- 

* 

jectlves. Advance answers to questions A through L, below, will 
provide a record of the rationale under which the evaluation was Ini- 
tiated: after- the- event answers, will record an assessment of the 

degree to which the expectations expressed by the Initial answers were 
fulfilled. As suggested In the dlsc.usslon of evaluation planning, the 
records provided by these answers should be available to other respon- 
sible offices, but they should not be passed around routinely. To- 
gether with the evaluation reports themselves, they provide the basic 
record of each evaluation. 

For each evaluation project, then, the following Information 
should be provided — for the record, before starting and after completing 
an evaluation — by the office Initiating the evaluation . These records 
should prove valuable to the staff members setting up and monitoring 
the project, to those budgeting for evaluation, to those Involved In 
review, and to those who ultimately will assess the evaluation effort 
Itself. 

A. WHERE IN THE TYPOLOGY OF PART II DOES THE EVALUATION FIT? 

For what decision-related purpose Is the evaluation being carried 
out? What Is being evaluated and for whom? It should be possible to 
code any evaluation, using the codes drawn from the typology, accord- 
ing to the answers to these questions. Such codes never fit every 
case completely comfortably (and, as has been noted, use will undoubtedly 
Indicate needed changes In the typology). For this reason, the major 
Information In the rationale documentation will be provided by verbal 
answers to the remaining questions. Nonetheless, codes are needed for 
purposes of summation In the planning and other processes. 



The discussion outlines laid out here under headings A through 
L need not be followed In detail. They are Intended only as guides. 





- 19 - 



B. WHAT DECISION OR DECISIONS MAY BE AFFECTED BY THE EVALUA TION REPORT 

As has been emphasized throughout, the purpose of evaluating a 
government program is to affect program or policy decisions; unless an 
evaluation project is designed for a particular decision, the chances 
are that the evaluation results will not change anything or even be 
seriously considered by a policymaker. Typically, the decision will 
be one of the following: 

1. Evaluation to Affect Allocation 

a. Go/No-go . A major program will very rarely be terminated be- 
cause of an adverse evaluation report. Once under way, programs develop 
a momentum of their own. The operators become entrenched, and the bene- 
ficiaries become an interest group. Under these circumstances, the 
most likely effect of adverse evaluation results on a major program is 
the discontinuation of a project or of a discrete activity within the 
program. When a no-go decision follows in the wake of an evaluation, 
it is apt to be because the results i^erve as merely a last (but perhaps 
crucial) piece of derogatory evidence. Similarly, decisions to under- 
take major programs are rarely based solely--or even largely--on favor- 
able results of pilot project evaluation. Rather, such results may be 
a contributory factor in a decision or supporting evidence for a decision 
that has, in effect, already been made. 

b. Other Increase or Decrease of Resources . Although an evalua- 
tion-based decision to terminate a major program is unlikely, if a 
program receives an unfavorable evaluation, its slice of the next budget 
is likely to be smaller than it would otherwise have been (although not 
necessarily smaller than it had been previously, if the program is on 
a growth curve) . Programs that receive favorable evaluations stand to 
gain. Too frequently, however, evaluations that measure overall program 
impact are designed to influence go/no-go decisions, providing little 
clear guidance on where to cut back an inefficient program or how to 
expand a seemingly effective one. Thus, if an evaluation is designed 
for use in a resource allocation decision, the prospective results 
should tell more than whether the impact (or the cost of achieving it) 



27 



- 20 - 



Is great or small. They should provide Information on what character- 
istics of the program cause the measured Impact or lack of impact. 

2. Evaluation To Improve Strategy 

a. Technique Choice . Program operators are likely to be relatively 
receptive to evaluations that examine alternative techniques applicable 
within their programs. However, if such evaluations have allocation 
implications, as evaluation of educational voucher programs does, for 
example, because of the implied possibility of allocating funds away 
from public school systems, they are as politically and bureaucratic- 
ally sensitive as any other evaluations. 

b. Technique Refinement . Evaluation can be useful in helping to 
shape programs by providing feedback to program managers on how tech- 
niques and processes can be improved. Results provided iteratively can 
facilitate the evolution of an effective program design. But the search 
for such marginal adjustments should not be used as an excuse to avoid 
hard decisions; the evaluator must take care that, in supporting sequen- 
tial minor decisions, he not become an accomplice to avoiding major 
ones. 

3. Compliance Control 

If the project in question is not an evaluation but a compliance 
control effort in fulfillment of part of the mission of the evaluation 
office, that fact should be noted and the requirement for the project 
should be stated. The remainder of Section IV of this Guide then be- 
comes irrelevant. 

4. Capability Building 

If the project is designed to facilitate or improve the conduct of 
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future evaluation studies by building a data base, by improving evalua- 
tion strategies and methodologies or by improving non-federal capabi- 
lities* the particular improvement to be achieved should be described. 
The question will be, what are the gaps, in data, in method, or in 
training, to be filled, and what are the needs of the evaluation office 
or other such offices — federal or other — for such capabilities? 

, i 28 
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0. HOW MIGHT POSSIBLE ALTERNATIVE RESULTS OF THE EVALUATION 
AFFECT THE DECISION? 



1, Evaluations 

For any evaluation, no matter what the type, the evaluation office 
should know in advance what kind of information is being sought what 
program aspects are being measured. Ordinarily, it will be possible 
to estimate the range of possible results. The central question, then, 
is: How are different possible results likely to affect the decisions 

discussed in Subsection B, above? Is a finding of "No impact" likely 
to be a factor in an allocation decision; how large an impact is likely 
to be of significant importance in the decision? Are certain kinds of 
results likely to be rejected out of hand, or are others likely to be 
misinterpreted? What is needed here is a contingency plan for the 
evaluation itself, laying out the decision implications of the range 
of possible results. 



2. Capability-building Projects 

How are the results of a project designed to increase capabilities 
likely to achieve this end? In what kinds of future evaluation projects 
are the new data likely to be used, and how? After these questions 
have been answered, the remaining questions in Section IV of the Guide 
will not be relevant. 

D. WHO WILL MAKE THE FINAL DECISION? 

Often "final authority" is not a very useful concept in planning 
evaluations. In the strictest sense it is lodged too high at the 
Presidential, agency head or Congressional committee chairman's level. 
For purposes of deciding whether and how to undertake a particular 
evaluation, the decisionmakers of interest are those who have the 
authority to make or are likely to recommend a decision bas^ sub- 
stantially on evaluation results. 




f < ^ 
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1. Which Decisionmaker 

a. In the Department of Health, Education and Welfare . The res- 
ponsibility within the government for considering the substantive merits 
of an Issue — as opposed to signing a directive — Is usually not precisely 
placed. Although most decision documents require the signature of a 
high-level executive, some — like refunding of a small demonstration 
program — are actually decided much further down. Others — like deter- 
mination of major program allocations — are decided In fact as well as 

In form at the executive level. Occasionally, It Is Impossible to 
determine In advance where the decision will be taken. Since effect 
on declslon-maklng Is central to mounting an evaluation. It Is Impor- 
tant to delineate the general focus of the decision as closely as 
possible, and to judge what factors are likely to determine who will 
finally make the decision. Will the nature of the evaluation results 
matter? What if the individuals involved change? 

b. Elsewhere In the Federal Government . As discussed below under 
(12), evaluations may be designed to assist Department officials to de- 
fend programs elsewhere — before the Office of Management and Budget or 
the Congress, for example. This is a perfectly normal and legitimate 
use of evaluation. It does not imply that the evaluation should be set 
up to produce biased answers; rather that a hypothesis of "favorable" 
results. If confiriried, can be useful at various levels of government. 
Deliberately biased evaluation, more often than not, has proven politi- 
cally counterproductive. If a program is controversial enough to need 
such evaluation results, it generally has suspicious opponents who can 
throw the results into question (as Indeed, they frequently do with the 
results of a completely objective evaluation). Therefore, the use of eval- 
uation for program defense has more limitations than Is sometimes recognized. 

c. Outside the Federal Government . For programs run locally, in 
all three areas of health, education and welfare, many decisions are 
taken by non- federal or non-public officials. Particularly in the case 
of evaluations designed to select or improve techniques, results may 
affect policy at these levels. If the techniques are applicable on a 
national or regional basis , the Federal government may be the appropriate 

30 
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level for Initiating evaluations whose results will affect decisions 
taken at lower levels. The evaluation of a particular form of sampling 
check of welfare eligibility, for example, may provide Information of 
Importance to welfare authorities In thousands of counties. If deci- 
sions are to be made at levels like this, the key to evaluation effec- 
tiveness lies In the ability to disseminate results widely to the many 
decisionmakers affected. (See F5, below.) 

2. Constraints on Decision 

In assessing the likelihood that evaluation results will affect a 
decision. It Is essential to consider what and who constrains a decision- 
maker's latitude for action. The Intention here Is not to suggest that 
every evaluation must fit precisely within these constraints. Such 
constraints are neither unchangeable nor permanent, and one function 
of evaluation Is to change the constraints. But failure to recognize 
the constraints that exist, and to understand how they app3.y to the 
particular evaluation In question, can cause the evaluation to become 
Irrelevant to any real policy decisions. Since at different levels 
decisionmakers are affected In different ways, an enumeration of con- 
straints must be specific to be useful. 

a. Presidency . What Is the President's position. If any, on the 
Issue? Is he hemmed In by past actions or statements? Are relevant 
White House Staff committed one way or another? What about 0MB? 

b. Congress . Are Influential members of Congress committed In a 
particular way for or against the program? How much have they Influ- 
enced past program decisions? how have they received past evaluation 
results? Are there important local district Implications? 

c. Partisan Politics . Does either party have a proprietary 
Interest in the program? Are there any major Ideological Issues? Is 
there strong organized bipartisan support or opposition? 

d. Administrative Politics . Is a decision likely to Impinge on 
the private preserve of another major agency? On a strong element 
within your own or another agency? Is there at present a clearly 
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establlshed lineup of advocates and adversaries within the bureaucracy? 
If so, how rigid Is It? 

e- Federal/State/Local Power Balance . Is an evaluation decision 
likely to affect the balance? If so, how delicate Is the balance? 

How Important Is It to the other fish the decisionmaker has to fry? 

f* Pressure Groups . Are any professional, beneficiary, or other 
groups effective as program advocates or adversaries? If so, how do 
they derive their power? How will they react to evaluation? 

3, Other Information Channels 

As a rule, decisionmakers do not rely on a single source of Infor- 
mation regarding program effectiveness. Information can be of high 
quality or low, can become available regularly or randomly, can range 
from highly subjective assessment to unprocessed data. Whatever the 
form, the existence and Impact of other Information channels should be 
taken Into account when evaluation decisions are made. In particular, 
how will other Information affect the credibility of evaluation find- 
ings? Is the Information source reasonably objective? Is the source 
an advocate or an adversary? Is the other source unimpeachable In the 
decisionmaker ' s eyes ? 

E. HOW WILL THE AVAILABILITY OF EVALUATION RESULTS 
MATCH THE DECISION TIMETABLE? 

1. When Will Decisions Be Made? 

Obviously, the decision timetables are Important to anyone who Is 
responsible for evaluation. If evaluation results are to matter, they 
must be In the hands of the decisionmaker when he Is deciding. This 
does not necessarily mean "the sooner, the better" — that the evaluator 
should attempt to have results available for the earliest significant 
decision. His first step nhould be to lay out the probable decision 
stream for the program to be evaluated. There will be differences In 
decision types (B, above) and levels (D) , as well as differences In 
timing. 

32 
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a. Major Decision Points . By definition there are few of these 
in the life of a program. They include such events as program initia- 
tion, program termination, major expansion or contraction, expiration 
of authorizing legislation, and Executive Branch reorganization. Their 
timing may, in turn, be related to other, more regular events, such as 
the State of the Union address. Presidential Messages, and the end of 

a fiscal year. Decisions made at these points can be affected by 
evaluation results as well as by many other factors. The evaluator 
needs to know when the major decision points fall. What might affect 
their timetable? What is the likelihood that the decision will be not 

to decide? 

b. Regular Periodic Decisions . Most of these decisions relate 
to the budget cycle. The President submits his budget to the Congress 
at a time specified by statute. Agency submissions to 0MB must conform 
to an annual budget calendar. Decisions to seek funds for a new pro- 
gram or to reallocate the agency's budget among programs are also made 
annually, usually after the agency's appropriation bill has been passed 
and signed by the President. Finally, a cleanup of funding decisions 
at the project level (including evaluation projects) occurs during the 
month of June, so that agencies will not lose funds that must be obli- 
gated before the end of a fiscal year. Which of these periodic deci- 
sions might be affected by evaluation results? How predictable is the 
occurrence of major decisions in the cycle? How far in advance? 

c. Incremental Decisionmaking . It is often difficult even in 
retrospect— to establish the time at which Important decisions on a 
program are made. Programs often grow along a path of very small de- 
cisions. Obviously, the technique- improvement class of evaluations is 
tailored to this pattern of decisionmaking. However, in some cases an 
evaluation may be undertaken precisely for the purpose of disrupting 
the incremental pattern. The Westinghouse Head Start evaluation, for 
example, was undertaken primarily in order to disrupt a pattern which 
tried to improve Head Start, without ever questioning its basic favor- 
able effect on children or the substantial fund allocation based on this 
presumed effect. The faults of this evaluation in failing to suggest 
improvement strategies, as well as its partial success in disrupting 
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the allocation assumptions, were largely caused by the evaluator’s 
desire to disrupt. To be effective, however, such disruption requires 
a good deal of skill on the part of the evaluation office and a thorough 
understanding of the decision environment. Why does the particular 
pattern of decisionmaking exist? Does evolution appear to have led to 
a satisfactory rate of improvement? What are the likely costs of giving 
up Incrementalism? What Is the most significant decision that can be 
made without disturbing the Incremental pattern? 

2. How Will the Evaluation Results Match This Decision Timetable? 

The timetable for results will be determined to some degree by 
the way in which the evaluation Is designed and the contract written. 
Hence, the evaluation office has considerable latitude for Influence. 

For some cases, the evaluator should follow "the sooner, the better" 
principle. For others, the cardinal principle will be "if you want It 
bad, you get It bad." It Is hard to formulate a general rule for set- 
ting up a timetable of evaluation results. The one worst decision on 
timing, however, is no decision — neither an early evaluation nor the 
necessary preparation for a later one. Because of the Incremental 
decisionmaking pattern of Head Start referred to above, no allocation 
evaluation had been mounted early in the program and no early plans 
laid for a later one, so that when a clear need for such an evaluation 
was manifest. It had to be done In a relatively quick and dirty way. 

a. Interim Findings . It Is almost always possible to obtain 
some results from an evaluation effort before the work is coj^plete. 

It Is not always wise to do so. The evaluation office should prudently 
assume that Interim evaluation results (as well as final ones) will 
leak out to advocates and adversaries alike. In the press and In the 
Congress, qualifying terms like "provisional," "incomplete," "raw," 
awH "not yet analyzed" tend to be dropped when Interim results are 
cited. Those who stand to be damaged by the evaluation may have some 
success in discrediting the whole evaluation on the basis of Interim 
results. Usually, decisions are made to proceed with social programs 
(such as the Family Assistance Plan) before they are fully field-tested. 
In such cases, even interim results can be useful In program design. 
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The evaluator needs to judge the probable soundness of interim results. 
How could they be used to good purpose and bad? How well can they be 
protected? Will they derogate from the final results? 

b. Final Results . The final results of most major evaluation 
projects should be targeted to a specific decision timetable. As a 
general rule« the sooner final results are available and can fit into 
the decision timetable, the better, and the near-term pernpectlve of 
most government executives will be a force in this direction. There 
are some Important exceptions to the rule, however. In some cases, 
longitudinal studies spanning several years may be preferable to studies 
whose final results will be available sooner. This is likely to be the 
case for a new program that appears sound but has many detractors on 
other grounds. In addition, during the first year or so, any program 
encounters start-up problems, and the results of overall Impact eval~ 
nations conducted early are unlikely to reflect the program’s realiz- 
able potential. Under such circumstances, any evaluation efforts offer- 
ing results in the formative year should focus on technique choice or 
improvement, with overall Impact best examined by means of longitudinal 
studies (which should be planned and begun early so that they can pro- 
vide timely results for allocation decisions later on, however). The 
problem is to determine how to fit the final results into the decision 
timetable for the most Impact. What wrong decisions might be made if 
final results slip in time? If they become available prematurely? 

c. Salvage Value . Predictions of decision timetables cannot be 
made with great certainty — even when there appears to be a regular 
cycle. Hence, it is worthwhile considering the "salvage value" of an 
evaluation. What if its results are not available when the targeted 
decisions are made? What if the particular issue it addresses never 
arises? What will the evaluation results reveal about the broad social 
problem area and about useful (or fruitless) ways of attacking it? 

F. WILL THE EVALUATION RESULTS BE READ AND UNDERSTOOD? 

Obviously, if evaluations are to affect decisions, someone with 
a major role in the decision process must read and understand them. 
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Thls does not necessarily mean reading and understanding an entire 
report. The higher the level of the decisionmaker, the less time he 
Is likely to spend on It. Yet, If he Is to use the results to defend 
a decision against critics, he will have to spend enough time to make 
his defense effective. It Is the responsibility of the evaluation 
office to supply material that Is Intelligible to the decisionmakers 
— the decisionmakers cannot be called upon to Interpret esoteric or 
Irrelevant material. The question, then. Is, will the report be read 
by; 

1. The Final Authority? 

Executives at this level rarely have the time to read and under- 
stand evaluation reports — or even summaries. However, If a high-level 
executive Is to be publicly credited with deciding on the basis of 
"objective" evaluation results, he should understand the conceptual 
basis for the evaluation and the general findings. This Is likely to 
take at least an oral briefing. 



2. By Those Who, In Effect, Decide? 

Usually more than one Individual may have authority to decide, 
and several may participate (see D, above). The evaluation report 
should be designed with all these potential decisionmakers In mind. 

They should be able not only to read and understand It, but also to 
defend che method and results against all but very technical criticism 
— e.g.. In a Congressional hearing or before a news conference with 
press specialists. 

3. By Those Who Advise and Inform the Decisionmaker? 

Normally, the evaluation results will be brought to a decisionmaker's 
attention by his own staff or other trusted advisers. The chief of the 
evaluation office Is likely to be among this group, or at least to be 
Influential with this group. (If not, the chances are that evaluations 
won't matter much.) At least one member of this group should understand 
the evaluation study well enough to defend It against a technical assault 
and keep the boss from using the results Incorrectly. 
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4. By External Individuals or Organiz ations? 

It is prudent to assume that any evaluation done by a contract 
evaluator (and many done Internally) will be read by Congressmen or 
Congressional staff members who have an Interest In the program. If 
It is of sufficient Importance, the evaluation will attract the Interest 
of the press. A Congressional committee member or staffer is likely to 
be interested In subjects under his Jurisdiction, a Congressman or re- 
porter In projects within his area. In recognition of the iron law of 
the leak (as well as the Moss Freedom of Information Act) , many agencies 
routinely make contractor evaluations public. But if the evaluations 
are important, they will be made public, whether routinely or not. 

They will thus be read externally; whether they will be understood is 

another question. 

5. By Anyone Who Should Use the Res ults for Program Purposes^ 

Many federally funded social programs are operated outside the 
federal structure—by state or local governments, or under private 
contract. If the evaluation Is supposed to affect decisions in such 
remotely run programs, as many are, a mechanism to disseminate the 
results is crucial to the effectiveness of the evaluation and to HEW's 
Important service role. The evaluation rationale should specify in 
some detail the means of dissemination of results. 

G. ARE THE EVALUATION RESULTS LIKELY TO BE MISI NTERPRETED? 

The concern here should be whether or not reasonable men are likely 
to Interpret the results differently or whether they can easily mis- 
interpret results for their own special purposes. (See F2a above.) 
Issues of methodology are obviously involved, but the focus here is on 
the non-technlcal readers who are unlikely to delve very deeply into 
methodological Issues or even to understand them. The evaluation may 
be methodologically sound, but the results may still be subject to mis- 
interpretation or misuse by someone who does not read deeply Into a 

report. 
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1 . Proxy Values 

Since It la rarely poaalble to obtain direct meaaurea of program 
outputs— meaaurea that coincide with atated program objectives — moat 
evaluation rsaulta are proxies or indirect measures. The objective of 
anti-poverty programs, for example, has been to raise the standard of 
living of those at the lower end of the income scale. The proxy mea- 
sure adopted for this, however, was the number of families crossing a 
purely arbitrary level of dollar income called the "poverty line." 

The problem, then, is that a proxy may carry one connotation for one 
person and quite a different one for another. The creation of the 
"poverty line," in the example, engendered endless debates over whether 
the arbitrary standard had been set at the "right" point. Because of 
this controversy, too little attention was focused on the number of 
people moving across the line as a general proxy Indicator with %iiilch 
to evaluate how well anti-poverty programs were achieving their complex 
Income- raising objectives. The evaluator's aim cannot be to renove 
ambiguity from proxies; that would be impossible. However, the eval- 
uator should consider whether a proxy might cause an unfortunate "knee- 
jerk" reaction. How clear is It which objectives the proxies serve? 

Are any of the proxies commonly used to represent an objective outside 
the program? Are any of the proxies likely to kindle the prejudices 
of someone %d.th a major role In the decision? 

2. Partial Results 

If the evaluation comprises a number of separate parts, the 
chances are that partial results will be considered. How damaging would 
this be? Would it lead to a wrong conclusion? Would the overall Impact 
of the evaluation be lost? 

3. Generalization 

Choosing a sample In a way that will lend broad significance to 
evaluation findings is a major methodological problem. Apart from this, 
however, the evaluator should consider whether a decisionmaker who pays 
little attention to methodology will construe the results either too 




38 



-31- 



broadly or too narrowly. Is there anything about the evaluation or 
the decision situation that suggests generalization beyond what is 
valid? For example, might enemies of a program seize upon an adverse 
evaluation of a project to discredit the whole program? Is an Important 
program advocate likely to reject valid generalization — say, because 
the sample did not cover the cities with "good projects"? 

H. IF AN EVALUATION-BASED DECISION IS MADE. CAN IT BE IMPLEMENTED? 

If an evaluation shows that certain kinds of counseling techniques 
are effective in a wide range of vocational rehabilitation programs, 
for example, and if the responsible authorities in Washington decide 
to adopt these techniques, will this be Implemented in the field? A 
simple directive may not do Itj dissemination of results of the eval- 
uation nay convince some local program administrators, but not others. 
Perhaps it cannot be done at all— —at least in the short run. In one 
sense, implementation is the decisionmaker's problem and not the eval- 
uator's. However, so many program decisions have run aground on imple- 
mentation that an evaluation design must pay some attention to the 
feasibility as well as the desirability of change. 

I, WHY DO DECISIONMAKERS WANT THE EVALUATION RESULTS? 

Decisionmakers do not always seek evaluations because they are in 
search of objective truth. Indeed, the state of the art of evaluation 
is not so high as to inspire unmitigated confidence in the results of 
even the best projects. Decisionmakers often have a policy goal in 
mind when they ask for an evaluation. The evaluation office should 
consider this in assessing the decision relevance of prospective eval- 
uations. Most situations will fall within the limits described below; 

1. "I want to Find Out What the Program Has Done" 

Devising programs that will alleviate major social problems is an 
uncertain business. The reason for evaluations is to find out what 
does or does not work. Ordinarily, however, decisionmakers have pre- 
conceptions about answers to the questions addressed by an evaluation. 
From the standpoint of the evaluation office, the best potential 
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consuner of results Is one who had not made up his mind; the second 
best Is one whose views are subject to change* A decisionmaker with 
strong jk priori views on all programs will be a good customer for 
evaluation only when It supports these views. Does the decisionmaker 
have a stake In the program or a strong view on It, as It now stands? 

Is he Inclined to look for new Ideas. How fertile Is the field for 
acceptance of fresh approaches? 

2* "I Want Some Ammunition with which to Defend/Attack the Program" 

As has been suggested, many evaluations have at least the Implicit 
purpose of providing a committed decisionmaker with ammunition to de- 
fend a program at a higher level--the Administrator before the Secretary, 
the Secretary before 0MB or the President or the Congress. Conversely, 
some approaches to social problems are generally discredited. Yet they 
persist In programs, and evaluations of such programs may be undertaken 
with the sole objective of adding weight to the argument for abolishing 
them, this fact being only thinly disguised. Neither of these Implicit 
objectives, of Itself, destroys the objectivity of an evaluation, al- 
though It may Impair Its credibility. However, there is cause to be 
concerned about the decision relevance of an evaluation if, first. Its 
major supporter Is someone who Is principally Interested In the support 
It will give him and, second, the outcome of the evaluation is uncer- 
tain. How unpredictable are the results? Does your decisionmaker- 
sponsor understand this as well as you do? How %^11 he react to the 
results if they do not support him? Is an end run feasible. If he 
tries to suppress them? Would It be worthwhile? 

J. WHO HA.S THE MANAGERIAL AND TECHNICAL COMPETENCE AND TIME 
TO DO THE EVALUATION? 

Sound evaluations of social programs demand Imaginative management 
and Imaginative use of professional skills. These qualities are often 
hard to find in combination; where they are found together, the person, 
government office or contractor tends to be overcommitted. These 
factors as well as credibility (discussed below) should determine the 
choice, or division of effort, between government and outside evaluators. 
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1. In-Government Evaluation 

a. Single Agency . Which of the agency's employees will do the 
work? Are they competent and sufficient In number? Does the team 
leader have the managerial and bureaucratic skill tc deal with probltms 

that may arise? 

b. TnrArA nencv Task Force . W^io will lead the task force, and 

how much time can he devote to the work? What kinds of representatives 
will the other agencies provide? How firm Is the claim on their time? 
Will parochial Interests of the agencies Interfere with effective use 
of their representatives' professional skills? Caution: An agency is 

usually unwilling to provide a significant amount of a. good man's time 
unless It wants to represent its particular Interest. 

c. Consult Ing Help . Will the government evaluation effort rely 
on significant help from outside consultants? Are consultants sure to 
be avalUble when needed? Will they function well under the guidance 
of the evaluation leader, or might they simply "do their thing t 

2. Contract Evaluation 

a. Competitive Bid . Are several contractors capable of design- 
ing an evaluation within carefully set terms of reference, of carrying 
It out, and of working with the In-house aUff? Is the In-house staff 
knowledgeable enough about the program and evaluation methodology to 
set terms of reference which will provide a firm basis for submission 

of competitive bids? 

b. Sole Source . Is only one contractor equipped to carry out 
work of the kind and quality desired by the contract office? Or Is 

the project design so complex that no competent contracting organlratlon 
will devote resources to producing a pre-contract evaluation design 
proposal that can be assessed on Its merits unless assured that It will 
be the final contractor? Can such a sole source decision be defended, 

legally and politically? 

c. If the evaluation Is to be contracted out. Is there sufficient 
lo-house «t«££ to design c»re£ully the terms o£ re£erence, to monitor 
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the evaluation while It is in process, and to interpret the results 
when it is completed? This may be the single most Important question 
in this guide. Evaluations do not carry themselves out, and even the 
best of contractors cannot substitute for in-house staff in the knowl- 
edge of the relevance of the evaluation to the decision process. 

Failure to provide adequate in-house control has almost always been at 
fault in the past, whenever evaluations have failed or turned out to 
be irrelevant. 

K. HOW CREDIBLE IS EVALUATION? 

Credibility has at least two dimensions — professional competence 
and freedom from bias. Both must be considered, whether the evaluation 
is done in-house or on contract. Professional competence is largely a 
question of reputation. If an organization — inside or outside of gov- 
ernment — is identified with sound analysis, the issue of credibility 
usually will not arise on grounds of competence. (Of course, the 
finished product will have to bear professional scrutiny.) But the 
question of bias will almost always arise independently whether the 
work is done in-house or contracted out. 

1. Credibility within the Executive Branch 

Can the evaluating office and the evaluator be considered suffi- 
ciently unbiased? This is a complex question. The Director of the 
National Institutes of Health may trust his evaluation office to tell 
him what is good or bad, or what should be changed in health research 
programs; the evaluation chief reports to him, not to the administrators 
of the separate programs being evaluated. But if the Director uses such 
an evaluation at a higher level to back up a program for which he is 
responsible, the Secretary of HEW may not accept the report as free 
from bias. To the Secretary, the Director of NIH is necessarily an 
administrator evaluating his own prograns — far from a disinterested 
party. Or if the Secretary does credit the Dir€Jctor*s lack of bias, 
the Director of the Office of Management and Budget may not. This sort 
of problem goes up and down the line. It is asymmetrical, since it is 
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the favorable evaluations that are most likely to be blessed by pro- 
gram administrators and least likely to be trusted by outsiders. 

2. Credibility on the Outside 

When an evaluation reaches the Congress and the public, will It 
be considered unbiased? The answer Is likely to be "No" for an eval- 
uation provided by the Executive Branch, for the reasons discussed 
above. This fact of life should not deter t.ie carrying out of evalua- 
tions. However, It does suggest that an evaluation designed primarily 
to convince the Congress of the viability of a program will seldom work. 



L. HOW DOES THE COST OF THE EVALUATION COMPARE WITH THE PROGRAM 
DOLLARS TO BE AFFECTED BY THE DECISION? 

If the two are even close, the evaluation should not be undertaken. 
Estimating program dollars to be affected by the decision Is difficult 
and enumerating evaluation cost Is far from trivial, although It must 
be done for contract work. The problem, as a whole. Is akin to that 
encountered by a speculative Investor matching his resources against 
Investment opportunities: The evaluation costs are analogous to the 

speculator's resources (his time, his limited ability to analyze thor- 
oughly all possible opportunities, his capital); the program dollars 
are analogous to the prospective return on the Investment. 

1. Evaluation Costs 

Contract dollars can be estimated Initially or put In as a celling. 
They can be tied down firmly before a decision Is made to go ahead. 

Costs In terms of evaluation office personnel resources are harder to 
estimate, because of the difficulty of predicting the bureaucratic and 
other problems an evaluation will run up against. It Is usually unnec- 
essary to attach a dollar figure to the office's personnel costs as 

*An exception to this rule might be when the evaluation has some 
general applicability and Is a way of getting started. 
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long as the total time constraint Is considered. However, where there 
Is a tendency to do In-house evaluations "to save money," It may be 
useful to rack up office personnel costs against the total staff budget. 

2. Program Dollars 

Obviously this estimate Is closely linked to types and levels of 
decisions to be effected and targeting of the evaluations to particular 
decision points. Like other "risk Investment" problems. It Is usually 
possible to Identify the several most likely outcomes, and It Is worth- 
while to attach some subjective probability to each "return." For an 
evaluation bearing In on allocation decision, for example, the likely 
range of budget variation, given alternative evaluation outcomes, 
should be estimated. (For a demonstration project, the relevant bud- 
gets are those that might fund a general adoption of the program being 
demonstrated. If the evaluation shows effectiveness. The cost of the 
evaluation may thus be large relative to the overall project cost.) 

In a technique-improvement evaluation, estimation should be relatively 
easy If the technique Is designed to be cost-saving. If It Is an 
effectiveness-increasing technique. It may be useful to estimate the 
size of the program to which an Improved technique will apply, although 
a direct comparison to costs will not be very meaningful. 



Evaluation experience In the Office of Economic Opportunity sug- 
gests a rule of thumb: a mid-level member of the evaluation staff 

(GS 12-14) should be assigned to monitor each $500,000-800,000 of 
annual evaluation work (two to five large projects). 
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V. THE CRITICAL PROGRAM SUMMARY 



Although the suggestions above for planning, for selecting, and 
for carrying out evaluations are centered on the role of the evaluation 
in the program decision process, they are nonetheless designed to struc- 
ture the evaluation process Itself. This section is concerned with pro- 
grams, and uses formal evaluation as one instrument among several in 
judging the programs. 

The Critical Program Summary la a matter of art, not of science. 

It depends heavily on judgment, the final judgment being that of the 
responsible head of the cognizant evaluation office. The Summary makes 
use of evaluation, but brings together data from many sources, not ex- 
cluding personal beliefs. Identified as such. The reason for the 
preparation of such summaries by evaluation officers Is that the very 
process of planning, selecting, supervising, and carrying out evalua- 
tions gives these officers unique viewpoints that can be of substantial 
assistance to decisionmakers deciding on program actions. 

The Critical Program Summary, then. Is a document prepared by each 
evaluation officer for each program for which he Is responsible, and 
submitted to both the administrative office to which the evaluation 
office reports and to the next higher evaluation echelon. The Summary 
should answer the following questions: 

A. WHAT IS THE PROGRAM IN QUESTION. AND TO WHAT PROBLEMS IS IT ADDRESSED ? 

What Is the problem? How many people does It affect (the "Universe 
of Need")? How does the program attempt to affect the problem? What 
portion of the Universe of Need does it cover? What is the legislative 
basis for the program, and the legislative history? 

B. WHAT DO WE NOW KNOW ABOUT THE PROGRAM? 

What is meant here is relatively hard knowledge. The question to 
be answered is, what knowledge has been provided us about the program 
from evaluations completed or partly completed? Such evaluations are. 
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o£ course, the major source of hard knowledge but other sources may be 
available too. For example, research reports discussing the problem 
to which the program Is addressed may be available to the evaluation 
offices; non-governmental efforts such as Journalistic reports, uni- 
versity research, and similar documents may be available. In any case, 
the answer laid out here should summarize everything that can be said 
about the program on the basis of relatively hard knowledge. 

C. WHAT DO WE THINK ABOUT THE PROGRAM? 

The answers to this question draw upon the hard knowledge just 
discussed, but this Is one source among many, all drawn together by 
the Informed judgment of the evaluation officer. In addition to hard 
knowledge, the evaluation officer may use quite Imperfect data, anec- 
dotal Information, logical judgments about what seems to be the case, 
even defensible personal beliefs. VHiat should be set forth here are 
judgments, then, but not unsupported judgments. They should be based 
on plausible arguments set forth explicitly, and they should summarize 
the evaluator to present his own Informed beliefs about the past and 
current courses of the program In question. 

D. WHAT ACTION DO WE RECOMMEND IN REGARD TO THE PROGRAM? 

To answer this question, the evaluation officer must play the 
role of decisionmaker. Granted that the evaluator Is not likely to 
bring to bear the full range of factors considered by the decisionmaker, 
he Is nonetheless a public official whose judgments In response to 
questions (A) and (B), above, should logically lead to policy recom- 
mendations. The evaluator thus should make his recommendations explicit 
here: what programs and projects should be promoted or eliminated, 

what allocations should be made, what techniques should be promoted? 

E. WHAT ADDITIONAL KNOWLEDGE WOULD LIKE TO HAVE ABOUT THE PROGRAM? 

Questions (A) through (D) are policy questions, the answers '.o 
which will necessarily be based on partially Informed judgment. In the 
very process of systematizing this judgment, however, gaps In knowledge 
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wlll be uncovered and spotlighted. In his answer to question (D), 
the evaluation officer has provided his best Judgment as an informed 
official as to what should be done. The answer to the present question 
vill identify further information that could cause him either to change 
his recommendation or to feel more comfortable in his initial Judgment. 

F. WHAT SHOULD BE DONE TO OBTAIN THIS INFORMATION? 

Here the evaluator returns to his initial and primary role. Con- 
sidering the information that he would like to have, what actions is 
he taking as an evaluator to obtain it and what actions might he take, 
or if he is not taking action, why not? What steps outside of the 
eval u ation process proper— research, experimentation, demonstration- 
ought to be taken by others to obtain the requisite information? 

The final document should be a bold one. Whereas evaluation 
itself must be relatively rigorous and eschew the highly conjectural, 
this document should be deliberately conjectural— the conjectures of 
a party with information and understanding that are different from 
and sometimes better than those of other parties to program decisions. 
The document that attempts to answer the questions of what should be 
done can be a major aid to those responsible for making these decisions 
—in some cases, perhaps, the greatest aid that can be provided by an 
evaluation office. 
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Apptndlx A 
DOCUMENT SUMMARY 



The various sections of this Guide suggest the following docu- 
aentstion: 

1. An annual evtluation plan, prepared by each evaluation office, 
and containing: 

a. A stateaent of strategic objectives, (page 11) 

b. A brief on each planned evtluation. (page 12) 

c. Four basic tables: 

(1) Planned evaluation activity: planning year, (page 15) 

(2) Actual evaluation activity: current year, (page 15) 

(3) Actual evaluation activity: last cosplete year, (page 15) 

(A) Previously prepared plan: last conplete year, (page 15) 

d. Three suaaary tables: 

(i; Evaluations initiated each year, (page 15) 

(2) Evaluations ongoing each year, (page 17) 

(3) Evaluation information becoming available each 
year, (page 17) 

2. A stateaent of rationale of each evaluation, prepared by the 
initiating office two tines, (pages 18-36) 

a. Before the evaluation is begun. 

b. After the evaluation is completed* 

3. A critical program susssary, prepared by each chief evaluation 
officer for each program for which he is responsible, prepared 
on a schedule dictated by the needs of program administrators. 

(pages 37-39) 
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